Spectral Convolution Networks

Previous research has shown that computation of convolution in the frequency domain provides a significant speedup versus traditional convolution network implementations. However, this performance increase comes at the expense of repeatedly computing the transform and its inverse in order to apply other network operations such as activation, pooling, and dropout. We show, mathematically, how convolution and activation can both be implemented in the frequency domain using either the Fourier or Laplace transformation. The main contributions are a description of spectral activation under the Fourier transform and a further description of an efficient algorithm for computing both convolution and activation under the Laplace transform. By computing both the convolution and activation functions in the frequency domain, we can reduce the number of transforms required, as well as reducing overall complexity. Our description of a spectral activation function, together with existing spectral analogs of other network functions may then be used to compose a fully spectral implementation of a convolution network. 1 Motivation Convolution networks are used for machine learning problems such as image classification, natural language processing, and recommendation systems [4, 3, 8]. They are represented as a graph of operators which are typically sequentially applied to some input image, eventually yielding a classification for that input. Convolution is an expensive operation which is replicated repeatedly within a single network. Computation of convolution in the frequency domain under the Fourier transform has been shown to provide a significant speedup versus traditional convolution network implementations [9, 4]. However, typically activation is run following convolution in practise, and previous researchers have been unable to find a spectral implementation for both. In this paper we describe spectral representations of the max activation function a(x) paying particular attention to computational complexity. Specifically, a(x) = max(0, x). The abbreviation ReLU is used throughout to designate the part of the network which computes this function.